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Abstract 



We report on a search for the flavor-changing neutral current decays B'^ — ^ 
fi'^fi~K~^ and — > jjL^^'K*^ using 88 pb~^ of data from pp collisions at 
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^/s = 1.8 TeV, collected with the Collider Detector at Fermilab. Finding 
no evidence for these decays, we set upper limits on the branching fractions 
B{B+ iJi+iJL-K+) < 5.2 X 10-6 and B{B° ji+jJi-K*^) < 4.0 x 10"^ at the 
90% confidence level. 

13.25.Hw, 13.30.Ce 
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In the Standard Model of electroweak interactions, the flavor-changing neutral current 
(FCNC) processes B+ n+n-K+ and S° n+n'R*'^ [0 are forbidden at tree level. At 
higher order, these decays occur through penguin and box diagrams with predicted branching 
fractions (0.3-0.5) x lO'^ for B+ ii+iJ,-K+ and (1.0-1.5) x 10"^ for S° ^ [g. 
Many deviations from Standard Model physics, such as charged Higgs bosons or anomalous 
W couplings, can increase these rates significantly 0. Because the spread in Standard 
Model predictions, due mainly to uncertainties in hadronic form factors, is well bounded, 
the observation of much larger branching fractions would be a signal for physics outside the 
Standard Model. 

To date, no observation of these decays has been reported. The CLEO collaboration 
has set the limits B{B+ fi+i2'K+) < 0.97 x 10"^ and B{B° fi+fi^K*^) < 0.95 x 10"^ 
at the 90% confidence level (C.L.). The CDF collaboration has previously searched 
for these decays in an 18 pb~^ data sample recorded in 1992-1993 and set the 90% CL. 
limits B{B+ /i+/i-K+) < 1.0 x 10"^ and B{B° /i+/i-ir*°) < 2.5 x 10"^ The result 
presented in this Letter is based on an 88 pb^^ combined data sample from the 1992-1993 
and 1994-1995 running periods and supersedes the earlier CDF result. 

Briefly, the method used in this search is as follows. The CDF trigger selects events 
containing two muon candidates, which we combine offline with and K*^ candidates 
to form B~^ and B^ candidates. Using the topologically identical resonant processes 
5+ J/^/jK+ {fi+fi~)K+ and 5° J/ipK*^ {ij,+fi-)K*° for normalization, we 
measure ratios of branching fractions, thus canceling most uncertainties due to B meson 
production and detection efficiency. Small acceptance and trigger efficiency differences due 
to the different decay kinematics are corrected with a Monte Carlo calculation. 

The CDF detector has been described in detail elsewhere 0. Detector components most 
relevant to this search are the muon chambers and charged particle tracking system. The 
muon system is composed of drift chambers outside the hadron calorimeter, allowing the 
reconstruction of track segments for penetrating particles in the pseudorapidity ^ range 
|?7| < 1. The minimum transverse momentum (pt) of detectable muons varies between 
1.4 and 2.2 GeV/c as a function of r], depending on the amount of material between the 
interaction point and the muon chambers. Charged particles are reconstructed in the central 
tracking chamber (CTC) and the silicon vertex detector (SVX) |P, both located inside a 
1.4 T solenoidal magnetic field. The CTC, covering |?7| < 1.1, is a cylindrical drift chamber 
containing 84 layers grouped into nine alternating superlayers of axial and stereo wires. 
The SVX consists of four layers of single-sided silicon microstrip detectors mounted at radii 
between 2.9 and 7.9 cm from the beam. It provides track measurements in the r-0 plane 
with impact parameter resolution (13 + AO/pt) /ini, where pt is in GeV/c. The impact 
parameter of a track is defined as the distance of closest approach to the beam in a plane 
transverse to the beam. 

CDF uses a three-level trigger system. The muon pair trigger requires the presence of two 
track segments in the muon chambers at the first level and matching CTC track candidates 
found by the central fast track processor (CFT) for one or both muon segments at the 
second level. The third-level trigger tightens the matching criteria using fully reconstructed 
CTC tracks. 

Muons inside the acceptance of the muon chambers are found by the first-level trigger 
with an efficiency that rises from about 40% at px = 1.5 GeV/c to a plateau value of 93% for 
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muon transverse momenta exceeding 3 GeV/c. The efficiency for finding the corresponding 
tracks with the CFT at level two rises from about 50% at = 2 GeV/c to about 96% for 
Pt > 2.3 GeV/c. If one muon does not have a matching CFT track a higher threshold 
is imposed for the other muon, leading to an efficiency that rises from about 50% at pt = 
2.6 GeV/c and reaches the same plateau value for > 3.1 GeV/c. The ffist type of trigger, 
requiring two CFT tracks, was not implemented during the 1992-1993 running period. A 
muon pair is called resonant when its invariant mass is close to the cc— resonances J/ip or 

Event selection is identical for resonant and non-resonant muon pairs. 

To reconstruct ii'^ii~K^ candidates, we combine the two muon candidates selected by 
the trigger with an additional charged track, assumed to be a kaon. The kaon track must 
satisfy pt > 2 GeV/c, and the B candidate must satisfy pr > 6 GeV/c. 

The reconstruction of the K*^ in the B'^ — >■ ^~K*^ process is via the K*^ K^tt~ 
decay. We require px > 0.5 GeV/c for each track, and to reduce random combinations, we 
require pt > '2 GeV/c for the K*^ candidate. To allow for the i^*(892)° width, we require 
the K*^ candidate mass to be between 796 and 996 MeV/c^. No particle identification is 
used, and each pair of tracks is considered as K~^n~ and 7i^K~, respectively. We choose 
the i^*7^*°-daughter mass assignment with invariant mass closer to the world-average K 
mass. 

An isolation requirement is a well established means of improving the signal to back- 
ground ratio in reconstructing B decays. We require isolation / > 0.6, where / is the trans- 
verse momentum of the B candidate divided by the scalar sum of the transverse momenta 



of the B and all other tracks in a cone ^J{Ari)^ + (A0)^ < 1.0 around the B momentum. 
The measured efficiency from our data for the requirement / > 0.6 is 92 ± 6% for B mesons 
with Pt > 6 GeV/c and |?7| < 1. 

A possible background for the rare decay search comes from B ^ J / ipK decays in which 
the kaon is misidentified as a muon and the muon is taken as the kaon instead. We apply a 
cut to remove candidates that are compatible with this hypothesis. In fact, no non-resonant 
candidate showed this ambiguity after all other selection cuts were applied. 

We require B daughters to be consistent with originating at a displaced decay vertex 
and inconsistent with originating at the beam line. All tracks forming the B candidate must 
include hits on at least three SVX layers, to ensure a good impact parameter measurement, 
and must miss the beam line with at least 2a signficance, where a is the estimated impact 
parameter resolution. The tracks are fit to a common decay vertex. The fit assumes that 
the B meson is produced at the beam line and constrains the B momentum to be parallel to 
its line of flight in the r-0 plane. The ffi quality must be acceptable (x^ < 20 with 4 degrees 
of freedom for ^'^ ^~K^ and 6 for fi~^fi~K*^) and the transverse decay length must exceed 
400 /im. The transverse decay length is the distance between the beam line and the B decay 
vertex measured in the r — plane. The beam spot of the Tevatron has a transverse size of 
approximately 25 /im (R.M.S.) and a mean that is measured with negligible uncertainty [ITU 



The impact-parameter and decay-length criteria are chosen to minimize the expected upper 
limit using the signal efficiency from a Monte Carlo calculation and a background estimate 
from the B sideband mass region 5.38-5.88 GeV/c^. 

Figures |I] and |^ show scatter plots of the n'^fi~K~^ and fi~^fi~K*'^ mass versus the /i"*"//" 
mass for candidates passing all requirements. The signal region for B mesons, whose half 
width is chosen to be twice the invariant mass resolution, includes reconstructed B meson 
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masses between 5.23 and 5.33 GeV/c^. 

The decays B+ J/xjj{ij')K+ and 5° ^ J/i){iP')K*^ followed by J/i){ip') n+fi' 
have product branching fractions two orders of magnitude larger than the Standard Model 
predictions for the non- resonant -B^ fi'^fi~K~^ and — > ^'^fi~K*^ decays. Candidates 
with an invariant mass of the muon pair in the regions 2.9-3.3 GeV/c^ or 3.6-3.8 GeV/c^ 
are therefore excluded from the rare decay search. Events that have dimuon masses within 
100 MeV/c^ of the J/ijj mass are used as the normalization sample. 

In the B mass region for fi'^fi~K~^ final states, we count 4 non-resonant candidates and 
count 122 J/i/j candidates. In case of the fi'^n~K*^ final state, no non-resonant candidate 
is found in the 5-mass region, while we count 76 J/tp candidates. The background for the 
resonant decays is estimated by counting events in the B mass sideband 5.38 — 5.88 GeV/c^ 
and scaling for the relative sizes of the B signal region and sideband regions. The estimates 
obtained this way are 0.4 events for B~^ J/ipK^ and 0.6 events for B^ J/ipK*^. 

The excess of resonant events in the sideband below the B signal region over the sideband 
above the B-mass is understood to come from incompletely reconstructed B — > J/ipX 
decays. The non-resonant excess below the B signal region is not understood. The high 
mass sideband, five times as wide as the signal region, contains 4 non-resonant fi'^fi~K~^ and 
2 non-resonant fi'^n~K*^ candidates. Note that simply extrapolating these sideband rates 
may underestimate the background in the signal region, because the high-mass sideband is 
used to optimize the selection criteria. 

Assuming all observed candidates in the signal region are signal events, we set upper 
limits on the numbers of observable FCNC decays using Poisson statistics and conclude that 
N{n+i^-K+) < 8.0 and N{n+n-K*°) < 2.3 at the 90% confidence level. 

We use a Monte Carlo simulation to calculate R^, the ratio of the total trigger and 
reconstruction efficiency for the non-resonant mode to that of the J/ip mode, including 
the effect of the restricted dimuon mass range in the case of non-resonant decays. In the 
Monte Carlo simulation, B decays are sampled from a phase space distribution and weighted 
according to the decay amplitude, using form factors and Wilson coefficients taken from 
Ref. ig. We find R,{^m+h-K+) = 0.79 and R,{^m+ K*'^) = 0.65. For observable numbers 
N{fi+fi-K^*^) and N{J/ipK^*^) of non- resonant and J/ if) decays, one obtains the branching 
fractions for the FCNC decays, 

B{B ^^^^^-K^*^) = x B{B J/z^i^W) x B{J /i^ - ^V")- 

Whereas the central values of R^ assume Standard Model couplings, distributions of kine- 
matic variables in the decays B — >■ can be sensitive to non-standard short-distance 
interactions |[ll|Jl^. This affects the trigger efficiency, the acceptance of the event selection, 
and the extrapolation over excluded dimuon mass regions. To study such model dependence, 
we varied the signs of the relevant Wilson coefficients at M-^y, as suggested in Ref. |Tl|, and 



found the change in efficiency to be much less than the change in the predicted branching 
fraction. No variation in R^ larger than 4% (12%) was found for ^~^fi~K~^ 

While this procedure does not cover the most general set of models, we conclude that the 
limits reported in this Letter, which are determined assuming Standard Model couplings, 
will change little when applied to many extended models. We do not treat model dependence 
as a source of error. 
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The impact of hadronic form factor uncertainties on was studied by replacing the 



form factors from Ref. by the two sets given in Ref. |13|. We use the observed change 
in R^, less than 4% (8%) for the K {K*) mode, as an estimate of the systematic error due 
to form factors. 

Dominant systematic errors are the 11% (14%) uncertainty on the product of world 
average values for the branching fractions B —>■ J/ipK^ {K*^) and J/ip ^ /^^/^~ [IHI; and 



the statistical uncertainty on the number of resonant events. All other sources of error 
are found to make much smaller contributions. We estimate the total systematic error, 
including hadronic uncertainties, to be 15% (20%) for [i^ ii~ K[K*), and include it in the 



determination of an upper limit on the branching fraction, using the method of Ref. |T5 
Assuming all observed candidates to be signal events we find 

B{B+ ^ i^+fi-K+) < 5.2 X 10^^ 

and 

B{B° li^li-K*^) < 4.0 X 10"^ 

at 90% confidence level. These are the strictest limits on these decay branching fractions to 
date. Assuming Standard Model predictions, the expectation is to observe approximately 
0.5 signal events in each channel. 

We thank the Fermilab staff and the technical staffs of the participating institutions for 
their vital contributions. This work was supported by the U.S. Department of Energy and 
National Science Foundation, the Italian Instituto Nazionale di Fisica Nucleare; the Ministry 
of Education, Science and Culture of Japan, the Natural Sciences and Engineering Research 
Council of Canada, the National Science Council of the Republic of China, the A. P. Sloan 
Foundation, and the Alexander von Humboldt-Stiftung. 
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FIG. 1. Scatter plot showing the ji^ n~ candidates. The hatched horizontal bands 

are the excluded regions around the J/^p and ijj' resonances. The signal region is between the 
vertical lines. The cross-hatched area in the upper left corner is kinematically forbidden. Entries 
with M{ji+ji-K^) < 5.23 GeV/c^ and dimuon masses in the J/V^ region can be attributed to 
incompletely reconstructed B — J j-^X decays. 
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